We compute the cross-correlation between a sample of 14,000 radio-loud AGN (RLAGN) with redshifts between 0.4 and 0.8 selected from the Sloan Digital Sky Survey and a reference sample of 1.2 million luminous red galaxies in the same redshift range. We quantify how the clustering of radio-loud AGN depends on host galaxy mass and on radio luminosity. Radio-loud AGN are clustered more strongly on all scales than control samples of radio-quiet galaxies with the same stellar masses and redshifts, but the differences are largest on scales less than 1 Mpc. In addition, the clustering amplitude of the RLAGN varies significantly with radio luminosity on scales less than 1 Mpc. This proves that the gaseous environment of a galaxy on the scale of its dark matter halo, plays a key role in determining not only the probability that a galaxy is radio-loud AGN, but also the total luminosity of the radio jet. Next, we compare the clustering of radio galaxies with that of radio-loud quasars in the same redshift range. Unified models predict that both types of active nuclei should cluster in the same way. Our data show that most RLAGN are clustered more strongly than radio-loud QSOs, even when the AGN and QSO samples are matched in both black hole mass and radio luminosity. Only the most extreme RLAGN and RLQSOs in our sample, with radio luminosities in excess of 10^26 W/Hz, have similar clustering properties. The majority of the strongly evolving RLAGN population at z~0.5 are found in different environments to the quasars, and hence must be triggered by a different physical mechanism.